<html>
<head>
  <title>01数据处理</title>
  <basefont face="微软雅黑" size="2" />
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="exporter-version" content="Evernote Windows/307027 (zh-CN, DDL); Windows/6.1.0 (Win32);"/>
  <style>
    body, td {
      font-family: 微软雅黑;
      font-size: 10pt;
    }
  </style>
</head>
<body>
<a name="1070"/>
<h1>01数据处理</h1>

<div>
<span><div><span>一 、让数据提供更好更精确的数据，<span style="color: rgb(227, 0, 0);">变化率</span>等</span></div><div>Thus, not all of the data you have is useful, and sometimes you need to do</div><div>further manipulation on your data to make it even more valuable before</div><div>feeding it through a machine learning algorithm.</div><div>比如：</div><div>Would you consider data that is simply the Open, High, Low, Close or data that is the Close, Spread/Volatility, %change daily to be better? I would expect the latter to be more ideal. The former is all very similar data points. The latter is created based on the identical data from the former, but it brings far more valuable information to the table.</div><div><br/></div><div>二、遇到null值怎么办</div><div>we fill any NaN data with -99999. You have a few choice here regarding how to handle missing data. You can't just pass a NaN (Not a Number) datapoint to a machine learning classifier, you have to handle for it. One popular option is to replace missing data with -99,999. With many machine learning classifiers, this will just be recognized and treated as an outlier feature. You can also just drop all feature/label sets that contain missing data, but then you're maybe leaving a lot of data out.</div><div><br/></div><div>三、parsing data 分解/析数据</div><div><br/></div><div><br/></div><div style="text-align: center;"><b style="font-size: 18pt; color: rgb(173, 0, 0); line-height: 1.45;">数据预处理的一系列方法</b><br/></div><div><span>    </span><br/></div><div><ul><li><b style="color: rgb(28, 51, 135); font-size: 14pt; line-height: 1.45;">自动填入缺失值</b><br/></li></ul></div><div><span>    <span>    <span>    <span>    1：使用一个全局常量填充缺失值，将缺失的属性值用同一个常熟代替</span></span></span></span><br/></div><div><span><span>    <span>    <span>    <span>    2：使用与给定记录 属于同一类的所有样本的均值和众数填充默认值，加入某数据集的一条属于a类的记录的A属性下存在缺失值，那么可以用该属性上属于a类的全部记录的平均值来代替该缺失值</span></span></span></span><br/></span></div><div><span><span>    <span>    <span>    <span>    3：用可能值来代替缺失值，可以用回归，基于推理的工具或者决策树归纳确定，加入利用数据集中其他顾客的属性，可以构造一颗决策树来预测相同属性的缺失值</span></span></span></span><br/></span></div><div><br/></div><div><ul><li><font style="font-size: 14pt; color: rgb(28, 51, 135);"><b>噪声数据的平滑方法</b></font></li></ul></div><div><span>    <span>    <span>    <span>    1：分箱， 通过考察邻居来平滑有序数据的值。平均值平滑，中值平滑，边界平滑。</span></span></span></span><br/></div><div><span><span>    <span>    <span>    <span>    2：聚类，将类似的值组织成群或者簇，离群点可以被检测，通过删除离群点来平滑数据</span></span></span></span><br/></span></div><div><span><span>    <span>    <span>    <span>    3：回归，通过回归方法（线性回归，非线性回归）让数据适合一个函数来平滑数据</span></span></span></span><br/></span></div><div><span><br/></span></div><div><ul><li><font color="#1C3387" style="font-size: 14pt;"><b>数据规范化三种方法</b></font></li></ul></div><div><span>    <span>    <span>    <span>    <span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">1：最小-最大规范化</span></span></span></span></span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">   <span>    <span>    <span>    <span>    <span>    </span></span></span></span></span> value = （当前值 - min）/（max - min）</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span>    <span>    <span>    <span>    </span></span></span></span>2：z-score规范化</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span>    <span>    <span>    <span>    </span></span></span></span> <span>    </span>   (1)：计算求得平均值avg，标准差t</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span>    <span>    <span>    <span>    <span>    </span></span></span></span></span>    (2)：规范化，value=（当前值 - avg）/t</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span>    <span>    <span>    <span>    </span></span></span></span>3：小数定标规范化</span></div><div><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span>    <span>    <span>    <span>    <span>    <span>    </span></span></span></span></span></span>通过移动属性f的小数点位置进行规范化，小数点移动位数依赖于f的最大绝对值</span></div><div><br/></div><div><font style="font-size: 14pt; color: rgb(28, 51, 135);"><b>数据离散化</b></font></div><div><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">1：监督离散化   离散化过程中使用类别信息</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">    基于熵的离散化方法，采用自定向下的分裂技术，在计算和确定分裂点时利用类分布信息</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><br/></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">2：非监督；离散化   离散化过程中没有使用类别信息</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">    (1)等宽离散化：将属性的值域划分成相同宽度的区间，区间的个数由用户指定，存在的问题是实例分布非常不均匀</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">    (2)等频离散化：将相同数量的对象放进每个区间，区间个数由用户指定</span></div><div style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;"><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">        等频方法的一种变体是近似等频离散化方法，其基本思想是基于数据近似服从正太分布的假设，对连续数据进行离散化</span></div><div><span style="font-family: 微软雅黑; font-size: 14px; line-height: 21px; widows: auto;">    (3)基于聚类的离散化方法</span></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div></span>
</div></body></html> 